Method for measuring network performance parity

ABSTRACT

A method for measuring network performance parity includes the steps of computing a call completion ratio for at least one network provider, and determining whether the call completion ratio passes a first test. These steps may further include computing a first call completion ratio for a first network provider, a second call completion ratio for a second network provider, a difference between the first call completion ratio and the second call completion ratio, and a variance for the difference. The method also includes the step of assessing whether a second test is determinate if the call completion ratio does not pass the first test, by determining whether the variance is greater than a variance cutoff. In addition, the method includes the step of assessing whether a call completion ratio passes the second test, if the second test is determinate, by determining whether the difference is greater than a threshold.

FIELD OF INVENTION

The present invention relates to a method for measuring the performanceof a telecommunications network. More specifically, it relates to amethod for measuring the performance parity of a telecommunicationsnetwork involving an Incumbent Local Exchange Carrier (“ILEC”) and atleast one Competitive Local Exchange Carrier (“CLEC”).

BACKGROUND OF THE INVENTION

Many Incumbent Local Exchange Carriers (“ILECs”) desire to gain entryinto the Inter-Local Access Transport Area (“LATA”) telecommunicationsbusiness within their own region. In order to gain such entry, an ILECmust demonstrate that it has been equitable in providing interconnectionservices to Competitive Local Exchange Carriers (“CLECs”). Typically,the interconnection services provided to CLECs are measured, at least inpart, through the network performance of the ILEC. In other words, ILECsare required to demonstrate that the interconnection services providedto CLECs are statistically comparable to the services the ILECs provideto their retail end users.

Presently, the methodology for measuring network performance parity withrespect to interconnection between ILECs and CLECs is based on busy-hourtrunk blocking statistics. Busy-hour trunk blocking relies on thepercentage of calls that are not completed (i.e., are “blocked”) on atrunk group final in its respective busy hour, where a busy hour isdefined to be the hour with the largest amount of trunk group “demand”(i.e., load). For more information on busy-hour trunk blockingstatistics, see “Trunk Traffic Engineering Concepts and Application,”Bellcore SR-TAP-000191, Issue 2, Dec. 1989, specifically incorporatedherein by reference.

There are both technical and historical reasons why networks havetraditionally been engineered to achieve a specified “busy-hourblocking” objective on trunk group finals. Until the mid-1980's (i.e.,AT&T divestiture), the Public Switched Telephone Network (“PSTN”) wasfocused on very few services (i.e., Plain Old Telephone Service(“POTS”), 800, and other voice-oriented applications) offered byessentially one company (i.e., AT&T and its sibling Bell OperatingCompanies). In addition. the traffic routing was almost alwayshierarchical, with routing changing very slowly, if at all. Also, thetrunk group (and network) “busy hours” were sharply defined (i.e., muchgreater load than any other hour of the day), and the associatedengineering methodologies, including Neal-Wilkinson Engineering, werewell-understood, based on commonly-available measurements, andimplemented in existing operations systems for telecommunication.

Today, in contrast to these conditions, the PSTN is a highly diversifiedarena, with many CLECs and wireless companies providing access services,including fast growing Internet access. The ILECs and CLECs may alsodiffer in their respective network or service “busy hours,” sometimes onshared groups, significantly complicating engineering and servicemanagement. While the busy-hour trunk blocking statistic is stillimportant in,engineering networks, it is intended to be used as anengineering tool, rather than a quality of customer service measure,since traffic blocked on a trunk group is often alternate-routed toanother trunk group and completed. Thus, busy-hour trunk blockingstatistics do not indicate the actual call disposition, in view of thefact that calls may be blocked on some trunk groups, yet eventuallycarried (i.e., completed) on other groups. This is especially true,given that network management controls are regularly utilizing“non-hierarchical” or “out-of-chain” routes to improve call completionsand customer service. In addition, marketing strategies by ILECs, CLECsand Inter-exchange Carriers (“IXCs”) are increasing and spreading thePSTN demands across the day, especially in “non-peak” hours, to create,essentially, “busy days” instead of “busy hours.” Thus, trunk blockagereports miss traffic that might be blocked during hours other than the“busy hour.” As a result, busy-hour trunk group blocking does notaccurately assess the quality of customer service.

Accordingly, it would be desirable to provide a new methodology formeasuring network performance parity that overcomes the disadvantages ofthe prior art and is innovative, accurate, fair and simple to use,especially in the multi-company arena of the present day PSTN. It wouldalso be desirable to provide a performance measure that is sensitive toa variety of outage and overload situations. In addition, it would bedesirable to have a parity metric that is based on classical statisticstheory and consistent with assumptions underlying other widely-acceptedtariffs and performance measures in the telecommunications industry.Moreover, it would be desirable to provide a new methodology formeasuring network performance parity that incorporates meaningfulmeasures of customer service and is based on call completion, ratherthan trunk blocking statistics. Finally, it would also be desirable tohave a parity metric that encourages cooperative business behaviors, bythe ILECs and each of their CLECs, that can result in high-qualityaccess services at reasonable costs.

SUMMARY OF THE INVENTION

The present invention provides a method for measuring networkperformance parity comprising the steps of computing a call completionratio for at least one network provider, and determining whether thecall completion ratio passes a first test. The method also comprises thestep of assessing whether a second test is determinate if the callcompletion ratio does not pass the first test. The method of the presentinvention further comprises the step of assessing whether the callcompletion ratio passes the second test if the second test isdeterminate.

In addition, the present invention provides another method for measuringnetwork performance parity comprising the steps of computing a firstcall completion ratio for a first network provider, a second callcompletion ratio for a second network provider, a difference between thefirst call completion ratio and the second call completion ratio, and avariance for the difference. The method also comprises the steps ofdetermining whether the first call completion ratio or the differencepasses an initial test, and whether the variance is greater than avariance cutoff if the initial test is not passed. Moreover, the methodcomprises the step of determining whether the difference is greater thana threshold if the variance is not greater than the variance cutoff.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a preferred network for ILEC,CLEC, and IXC telecommunications.

FIG. 2 is a flow chart illustrating a preferred embodiment of a methodof the present invention with a parity test for measuring networkperformance parity.

FIG. 3 is a graphical representation of three examples illustratingpossible outcomes for the parity test and method of FIG. 2.

FIGS. 4A-4B are graphical representations of five cases illustratingadvantages of the parity test and method of FIG. 2 over the prior art.

FIG. 5 is a flow chart illustrating an exemplary embodiment of theparity test and method of FIG. 2.

FIG. 6 is a graphical representation of relationships between designparameters for the parity test and method of FIG. 5.

FIG. 7 is a graphical representation of the parity metric of the methodof FIG. 5.

FIG. 8 is a graphical representation of two examples illustrating a FAILoutcome for the parity test and method of FIG. 5.

FIGS. 9A-9B are graphical representations of three examples illustratinga PASS outcome for the parity test and method of FIG. 5.

FIG. 10 is a graphical representation of two examples illustrating anINDETERMINATE outcome for the parity test and method of FIG. 5.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Turning now to the drawings, FIG. 1 shows a preferred network 5 for ILECand CLEC local, intra-LATA (i.e., toll required), and inter-LATA or IXCcommunications, with comparable network arrangements (“CNAs”) for theILEC and the CLEC. The preferred network 5 comprises a first ILEC EndOffice (“EO”) 10, an ILEC tandem 20, an IXC provider 30, a CLEC EO 40,and a second ILEC EO 50. It should be understood, however, that thepreferred network 5 may have any number of different configurations,with more or fewer components than those identified and describedherein, depending on the application, as well as operating and consumerpreferences. For example, the first ILEC EO 10 and the second ILEC EO 50may be combined to form one overall ILEC EO, rather than two separateILEC EOs, as shown in FIG. 1.

The first ILEC EO 10 is connected to the CLEC EO 40 through a pair ofhigh usage (“HU”) trunk groups A, A′. These trunk groups A, A′ may beeither primary (i.e., give overflow, but do not receive overflow) orintermediate (i.e., give and receive overflow) HU-type trunk groups.Since both types of HU trunk groups give overflow, calls not carried onthese trunk groups may overflow to alternate trunk groups. Preferably,the trunk group A is provided by the ILEC for local and intra-LATA CLECcommunications from the first ILEC EO 10 to the CLEC EO 40. The A′ trunkgroup is preferably provided and administered by the CLEC for local andintra-LATA CLEC communications from the CLEC EO 40 to the first ILEC EO10.

As shown in FIG. 1, the first ILEC EO 10 is also connected to the secondILEC EO 50 through a direct final (“DF”) trunk group F. Since the trunkgroup F is a DF, the trunk group F does not automatically give overflow,and therefore, calls may be blocked. The trunk group F is preferablyprovided by the ILEC for local and intra-LATA ILEC communicationsbetween the first ILEC EO 10 and the second ILEC EO 50. The second ILECEO 50 is also connected to the ILEC tandem 20 through an alternate final(“AF”) trunk group B. Since the trunk group B is an AF, the trunk groupB may receive overflow, but does not automatically give overflow, andtherefore, calls may also be blocked. Preferably, the trunk group B isprovided by the ILEC for local and-intra-LATA ILEC communications, aswell as inter-LATA or IXC communications, between the second ILEC EO 50and the ILEC tandem 20, which is connected to the first ILEC EO 10, theIXC provider 30, and the CLEC EO 40.

In addition, the first ILEC EO 10 is connected to the ILEC tandem 20through an alternate final (“AF”) trunk group B′. Since the trunk groupB′ is an AF, the trunk group B′ may receive overflow, but does notautomatically give overflow, and therefore, calls may also be blocked.Preferably, the trunk group B′ is provided by the ILEC for local andintra-LATA ILEC/CLEC communications, as well as inter-LATA or IXCcommunications, between the first ILEC EO 10 and the JLEC tandem 20,which is connected to the second ILEC EO 50, the IXC provider 30, andthe CLEC EO 40.

As shown in FIG. 1, the CLEC EO 40 is connected to the ILEC tandemthrough a pair of AF trunk groups C, C′. As AFs, the trunk groups C, C′may receive overflow, but do not automatically give overflow, andtherefore calls may be blocked. Preferably, the trunk group C isprovided by the ILEC for local and intra-LATA CLEC communications fromthe ILEC tandem 20 to the CLEC EO 40. The C′ trunk group is preferablyprovided and administered by the CLEC for local and intra-LATA CLECcommunications from the CLEC EO 40 to the ILEC tandem 20.

The CLEC EO 40 is also connected to the ILEC tandem through anintermediate final (“IF”) trunk group D, an AF trunk group E, and an HUtrunk group G. As an IF, the trunk group D both gives and receivesoverflow, while the trunk group E, as an AF, receives overflow, but doesnot automatically give overflow. As a result, calls may be blocked ontrunk group E, but not on trunk group D. Preferably, the trunk group Dis provided by the CLEC for inter-LATA or IXC CLEC communicationsbetween the ILEC tandem 20 (and thus the IXC provider 30) and the CLECEO 40. Similarly, the trunk group E is preferably provided by the ILECfor inter-LATA or IXC CLEC communications between the ILEC tandem 20(and thus the IXC provider 30) and the CLEC EO 40. Like the trunk groupsA, A′, the trunk group G may be either a primary (i.e., gives overflow,but does not receive overflow) or an intermediate (i.e., gives andreceives overflow) HU-type trunk group that may overflow to alternatetrunk groups.

The hierarchical traffic routes for the preferred network 5 will now bedescribed with reference to FIG. 1. With respect to the first ILEC EO10, local and intra-LATA CLEC communication traffic is routed betweenthe first ILEC EO 10 and the CLEC EO 40 along the HU trunk groups A, A′.Those calls that cannot be carried (i.e., are blocked) by the trunkgroups A, A′, overflow (i.e., are alternate-routed) to the trunk groupsB′, C, C′ according to a predetermined hierarchical rerouting plan. Forinstance, if a local or intra-LATA call from the ILEC EO 10 is blockedon the trunk group A, then the call overflows and is alternate-routed tothe CLEC EO 40 through the trunk group B′, the ILEC tandem 20, and thetrunk group C. Likewise, if a local or intra-LATA call from the CLEC EO40 is blocked on the trunk group A′, then the call overflows and isalternate-routed to the ILEC EO 10 through the trunk group C′, the ILECtandem 20, and the direct connection between the first ILEC EO 10 andthe ILEC tandem 20, i.e., the trunk group B′.

With respect to the CLEC EO 40, inter-LATA or IXC CLEC communicationtraffic is routed between the CLEC EO 40 and the ILEC tandem 20, whichis connected to the IXC provider 30, along the HU trunk group G. Thosecalls that cannot be carried (i.e., are blocked) by the trunk group G,overflow (i.e., are alternate-routed) to the IF trunk group D, oralternatively to the AF trunk group E, according to a predeterminedhierarchical routing plan. For instance, if an inter-LATA or IXC callfrom the CLEC EO 40 is blocked on the trunk group G, then the calloverflows and is alternate-routed to the ILEC tandem 20 through the IFtrunk group D. Alternatively, if the call is further blocked on the IFtrunk group D, the call may be further routed to the ILEC tandem 20through the AF trunk group E. The same is true for inter-LATA or IXCCLEC communication traffic from the IXC provider 30 to the CLEC EO 40through the ILEC tandem 20.

While uncompleted calls on trunk groups B, B′, C, C′, E, and F (i.e., anAF or DF trunk group) do not automatically overflow, and therefore maybe blocked, network management may determine a way to ultimatelycomplete the “blocked” calls. By using conventional non-hierarchical(out-of-chain) routing, network management may be able to complete some,if not all, of the calls blocked in the hierarchical routing scheme.Accordingly, a measure of call completion, rather than trunk groupblocking, more accurately reflects the quality of customer serviceprovided by the ILEC to its retail end users, the CLEC, and thecustomers of the CLEC.

As a result of the above hierarchical traffic routing, CNAs are formedamong the trunk groups for the ILEC and CLEC communications. Forinstance, ILEC trunk group F is comparable with trunk groups C and C′carrying CLEC traffic, and ILEC trunk groups B and B′ are comparablewith trunk groups D and E carrying CLEC traffic. In other words, thetraffic mix and function of trunk groups F and B, B′ are similar (i.e.,comparable) to the traffic mix and function of trunk groups C, C′ and D,E, respectively.

FIG. 2 shows an overview of a preferred embodiment of a method 100 ofthe present invention that uses a parity test (or parity metric) formeasuring relative network performance parity. Preferably, the method100 of the present invention is used to measure the performance parityof the CNAs described above. By measuring performance parity among CNAs,the method 100 of the present invention is able to maximize the accuracyand usefulness of its results. It should be understood, however, thatthe method 100 of the present invention may be used to measure theperformance parity of other CNAs or network arrangements that are notcomparable, or a combination of CNAs and non-comparable networkarrangements, and the method 100 of the present invention is not limitedto measuring the performance parity of only the CNAs previouslydescribed.

As shown in FIG. 2, the method 100 of the present invention begins withStep 101, wherein the call completion statistics for one or more trunkgroups are computed for a given period of time. Preferably, the callcompletion statistics for each CNA of the ILEC and the CLEC are computedas a percentage or ratio and analyzed together on a monthly basis. Themonthly basis may be a business month, covering aggregated daily callcompletion statistics for each day in a 20-day period.

The daily call completion percentage or ratio (“CR”) for one or moretrunk groups (e.g., a CNA) may be computed with the following equation:${{CR} = \frac{\text{call~~attempts} - \left( {\text{blocked~~calls} - \text{successful~~reroutes}} \right)}{\text{call~~attempts}}},$where CR may be stated as a fraction or a percentage. In this preferredequation, “call attempts” represents the total number of call attemptsfor a trunk group or set of trunk groups, “blocked calls” represents thetotal number of calls blocked by the trunk group or set of trunk groupswithin a hierarchical rerouting scheme, and “successful reroutes”represents the total number of calls successfully completed throughrerouting outside of the hierarchical routing scheme. Preferably, the“call attempts” and “blocked calls” include only those call attempts andblocked calls, respectively, on the hierarchical or “in-chain” finaltrunk groups. As an example, if 1000 call attempts were made on one ormore final trunk groups, with 100 calls being blocked within thehierarchical routing scheme, but 70 calls being successfully completedand rerouted outside of the hierarchical routing scheme, the CR for theone or more trunk groups would equal 0.97 or 97%. It should beunderstood that if there were no “sucessful reroutes,” then the CR isessentially the same as the complement of the “blocked calls” (i.e.,1-B), which is discussed in more detail below.

The next step of the method 100 is Step 102, which tests whether the CRfor the CLEC trunk group or set of trunk groups passes an initialacceptability screen. In Step 102, the CR for the CLEC trunk group orset of trunk groups is compared with the CR for the CNA of the ILEC. Ifthe CLEC CR exceeds or is consistent with that of the ILEC, thenperformance parity is achieved and the test declares a PASS 103, asshown in FIG. 2. The CR may also be compared to CRs based on other testsor factors, such as a CR based on known engineering objectives for theCLEC trunk group or set of trunk groups. For instance, if the CLEC CRexceeds or is consistent with a CR based on known engineeringobjectives, then performance parity is also achieved and a PASS 103 isdeclared.

Once a PASS 103 is declared by the initial acceptability screen test ofStep 102, network performance parity is achieved, and the method 100 ofthe present invention need not proceed any further. If a PASS 103 is notdeclared by the initial acceptability screen test of Step 102, however,because the CR for the CLEC trunk group or set of trunk groups does notexceed or is not consistent with the CR of the CNA of the ILEC oranother CR, such as the CR based on known engineering objectives, thenfurther testing must be done in Steps 104 and 105. In Step 104, adetermination is made of whether the parity test of the presentinvention is capable of making a decision (i.e., is determinate). If thevariance, σ², of an average difference between the ILEC and CLEC CRs, δ,is so large that the test design parameters cannot be met, i.e., the“noise” exceeds the “signal,” the parity test is unable to make adecision (i.e., is not determinate) with respect to the performanceparity between the CLEC and the ILEC CNAs. As a result, the parity testyields an INDETERMINATE outcome 106, as shown in FIG. 2. TheINDETERMINATE outcome 106 may require an investigation by either theCLEC, the ILEC, or both, into the causes of the large variance todetermine the proper course of action that should be taken in response.Such an investigation may reveal that the large variance was caused byhighly volatile or unforecasted demands, network outages or overloads,marketing and sales campaigns, or other unusual events beyond thecontrol of the ILEC and/or CLEC, rather than insufficient networkresources. Accordingly, investigating the causes of the large variancemay prevent the unnecessary provisioning of extra (or “reserve”) trunksand expenditures by the ILEC and/or the CLEC.

On the other hand, if the parity test of the present invention is ableto make a decision (i.e., is determinate) with respect to theperformance parity between the ILEC and CLEC CNAs (i.e., the variance issmall enough), the parity test is run to see if it “passes” in Step 105.In Step 105, the parity test compares an average difference between theILEC and CLEC CRs, δ, with a threshold, T, where T depends on both avariance, σ², of the average difference and several test designparameters, including a “false alarm (Type I)” objective, α, a “miss(Type II)” objective, β, and a null or alternate hypothesis of theaverage difference, Δ. Preferably, α, β and Δ are chosen to beconsistent with the assumptions in known equal-access blockingperformance tables. The parity test of the present invention isdescribed in more detail below with reference to specific examples ofits implementation. The parity test “passes” in Step 105 when theaverage difference between the ILEC and CLEC CRs, δ, is less than orequal to the threshold, T. If the parity test “passes,” then a PASS 103is declared and the method 100 of the present invention is completed. Incontrast, if the average difference between the ILEC and CLEC CRs, δ, isgreater than the threshold, T, the parity test does not pass in Step105, and a FAIL 107 is declared. Similar to the INDETERMINATE outcome106, a FAIL 107 preferably requires an investigation of its causes byeither the ILEC, the CLEC, or both, to determine the cause of thedifference in performance, since the ILEC is only responsible for causeswithin its control. An investigation by the ILEC and/or the CLEC helpsto expedite reasonable corrective actions, consistent with the knownpolicy of proactively servicing network problems. Such reasonablecorrective actions may include increasing network capacity by the ILECand/or the CLEC.

FIG. 3 shows three examples illustrating the possible outcomes for theparity test and method 100 of the present invention. In Example 1 ofFIG. 3, the ILEC CR is near 100% for an entire 20-day business month,while the CLEC CR is just below 80% for the same month. This relativelylarge average difference (i.e., over 20%) between the ILEC and CLEC CRs,δ, is illustrative of the FAIL 107 outcome, with no performance paritybeing achieved. In Example 2 of FIG. 3, however, the ILEC CR is about99% for an entire 20-day business month, while the CLEC CR is about 98%for the same month. This relatively small average difference (i.e., 1%)between the ILEC and CLEC CRs, δ, is illustrative of the PASS 103outcome, with performance parity being achieved. Finally, in Example 3of FIG. 3, the ILEC CR is near 100% for an entire 20-day business month,while the CLEC CR is also near 100% for most of the same month, exceptfor a few days where the CLEC CR dropped down eventually to less than60%. This relatively large variance, σ², in the CLEC CR is illustrativeof the INDETERMINATE outcome 106, with the performance parity outcomeincapable of being determined.

FIGS. 4A-4B illustrate some of the advantages of using the method of thepresent invention based on a CR measure compared to the traditionaltrunk group busy-hour blocking measure. In each of the five cases shownin FIGS. 4A-4B, the loads and associated blocking statistics on acollection of 100-trunk final groups were replicated for each hour ineach day of a 20-day business month. Each of the five cases shown inFIGS. 4A-4B illustrates call completion results for ten differentsamples. In order to properly illustrate the performance measuringeffects of the present method based on CRs compared to the trunk groupbusy-hour blocking of the prior art, offered loads of about 100 Erlangswere used for each of the five cases. As is well-known, an Erlang is aunit of traffic demand, with one Erlang being the maximum load that canpossibly be handled by one trunk in any given period of time (typicallyone hour). For instance, a trunk group with 100 call attempts during anhour and average call holding time of 1/20 of an hour (i.e., 5 minutes)would have 100×( 1/20)=5 Erlangs of offered traffic in that hour.

In each of the five cases shown in FIGS. 4A-4B, a trunk group's averagetotal CR for each day of a 20-day period is graphed together with thetrunk group's busy hour (“BH”) call completion ratio, 1-B, where B isthe daily, preferably unweighted, blockings in the hour with thegreatest offered loads over the same period of time. The “lower” curvein each graph corresponds to the performance measure (i.e., CR or 1-B)that is more sensitive to overloads for that case. For example, as shownin case 1 of FIG. 4A, both the CR and 1-B measures are comparable andreflect generally good service, since they both average in excess of 99%and differ by only about 0.25%. Similarly, in case 2 of FIG. 4A, boththe CR and 1-B measures are roughly comparable, since they both averageabout 88% and differ by only about 0.5%. Both the CR and 1-B measures ofcase 2, however, indicate service degradations due to overloads inseveral hours throughout the day (i.e., not just the BH). The servicedegradations may trigger action on behalf of the ILEC and/or the CLEC,such as augments or increases in network resources, depending on thenature of the service degradation.

Case 3 of FIG. 4B illustrates the effect of a distinct, high-load BH onthe CR and 1-B measures. In case 3, the 1-B measure averages about 88%,3% lower than the CR at about 91%. The CR measure in case 3 is somewhatless sensitive than the 1-B measure to the distinct, high-load BH,although both the CR and 1-B measures would indicate service degradationand probably trigger action by the ILEC and/or the CLEC.

Case 4 of FIG. 4B illustrates the effect of an overloaded BH that hasshifted or spread out on the CR and 1-B measures. In case 4, the 1-Bmeasure averages about 91%, 5% higher than the CR at about 86%. Unlikecase 3, the 1-B measure in case 4 is far less sensitive than the CRmeasure to the BH shift or spread. In fact, since the BH has shifted orspread out, the 1-B measure has essentially missed the resulting networkoverload. Similarly, in case 5 of FIG. 4B, the 1-B measure misses asingle off-peak (i.e., non-BH) overload of 1000 Erlangs, while the CRmeasure captures the overload. In case 5, the 1-B measure averages about94%, 1.5% higher than the CR at about 92.5%. Like case 4, the 1-Bmeasure in case 5 is less sensitive than the CR measure to the singleoff-peak overload of 1000 Erlangs.

FIG. 5 shows an exemplary embodiment of the method and its parity test(or parity metric) of the present invention (see FIG. 2) for comparingILEC and CLEC CNAs. While only one CNA and its CR is used for each ofthe ILEC and CLEC, it should be understood that these CNAs may wellrepresent multiple CNAs or a subset of the available ILEC and CLEC CNAs.The method 200 shown in FIG. 5 begins with Step 201, wherein the averageCR for the ILEC (“CR_(ILEC)”) and the CR for the CLEC (“CR_(CLEC)”) arecomputed for a given period of time, such as a 20-day business month.The sample difference, δ, and the sample variance, σ², of the sampledifference, is also computed in Step 201. The sample difference, δ, maybe computed by subtracting the CR_(CLEC) from the CR_(ILEC) (i.e.,δ=/CR_(ILEC)−CR_(CLEC)). The sample variance, σ², may be computed withthe following equation:$\sigma^{2} = \left( {\frac{\sigma_{ILEC}^{2}}{n_{ILEC}} + \frac{\sigma_{CLEC}^{2}}{n_{CLEC}}} \right)$In this equation, σ_(ILEC) ² and σ_(CLEC) ² are the variances for theILEC and CLEC CNAs, respectively. Also, the N_(ILEC) and the N_(CLEC)are the time periods or number of business days used in the calculationof the CR_(ILEC) and CR_(CLEC), respectively. A key attribute of thesample variance, σ², is that it will be small for CNAs with veryconsistent performance measures. If the sample variance, σ², is smalland the sample difference, δ, is also small, then parity between theILEC and CLEC CNAs has generally been achieved, as described in moredetail below. On the other hand, if the sample variance, σ², is smalland the sample difference, δ, is large, then parity between the ILEC andCLEC CNAs has generally not been significantly achieved, as alsodescribed in more detail below.

The next step of the method 200 is Step 202, which tests whether theCR_(CLEC) is less than an initial acceptance-screen value. As shown inFIG. 5, this value is preferably 97%, since 97% is consistent withdocumented and accepted engineering thresholds. For instance, accordingto well-accepted practices, a null hypothesis of 1% average blocking andan alternative hypothesis of 5% average blocking are common engineeringobjectives for trunk group finals. The alternative hypothesis of 5% isconsistent with time-tested (i.e., nearly 50 years) of industryexperience that small differences in observed blocking (i.e., 1-3%) donot trigger customer complaints, but that differences on the order of4-5% or more should be avoided. Almost any reasonable statistical testattempting to distinguish these two objectives would choose a thresholdroughly halfway between the two, or at about 3%, thereby leaving anacceptable engineering objective for CRs of 97%. The same conclusion wasreached in the Committee T1 Report No. 11 (June 1991) and Bellcore SRSTS-000317 (September 1990), both of which are specifically incorporatedin their entirety herein by reference.

In Step 202, if the CR_(CLEC) is equal to or exceeds 97%, thenperformance parity is achieved and a PASS 203 is declared, as shown inFIG. 5. If the CR_(CLEC) is less than 97%, however, then the method 200continues with Step 204, where the sample difference, δ, is tested tosee if it is positive, negative, or zero. In Step 204, if the sampledifference, δ, is less than zero (i.e., negative) or equal to zero, thenperformance parity is achieved and a PASS 203 is declared, as shown inFIG. 5, since the CR_(CLEC) is greater than, or at least equal to, theCR_(ILEC). On the other hand, if the sample difference, δ, is greaterthan zero (i.e., positive), then performance parity may or may not beachieved, since the CR_(CLEC) is less than the CR_(ILEC).

Once a PASS 203 is declared by either Step 202 or Step 204, networkperformance parity between the ILEC and CLEC is achieved, and the method200 of the present invention need not proceed any further. If a PASS 203is not declared by either Step 202 or Step 204, however, then furthertesting must be done in Steps 205, 206, and 208. In Step 205, athreshold, T, for the sample difference, δ, and a cutoff variance,σ_(cut) ², for the variance, σ², are computed. In Step 206, thevariance, σ², is compared to the variance cutoff, σ_(cut) ², in order todetermine whether the parity test or metric of the present invention iscapable of making a decision (i.e., is determinate). Equivalently, therespective standard deviations (“SD”), σ and σ_(cut), can be compared.As shown in FIG. 7, the preferred SD cutoff, σ_(cut), is 1.198.Accordingly, if the SD, σ, is greater than the SD cutoff, σ_(cut), of1.198 (i.e., to the right of σ_(cut) in FIG. 7), the parity test isunable to make a decision (i.e., is not determinate) with respect to theperformance parity between the CLEC and the ILEC CNAs. Consequently, theparity test yields an INDETERMINATE outcome 207, as shown in FIGS. 5 and7, which is identical to the INDETERMINATE outcome 106, with the sameresults. As explained below, with a variance, σ², that is so large thatthe test design parameters (i.e., α, β, Δ) cannot be met, the paritytest or metric is unable to make a decision (i.e., is not determinate)with respect to the performance parity between the CLEC and the ILECCNAs because the “noise” exceeds the “signal.” On the other hand, if theparity test of the present invention is able to make a decision (i.e.,is determinate) with respect to the performance parity between the ILECand CLEC CNAs (i.e., the SD, σ, is less than the preferred SD cutoff,σ_(cut), of 1.198), the parity test is run to see if it “passes” in Step208.

As previously set forth, large values of δ, or large values of anexpected value function of δ, E(δ), might suggest a possible lack ofparity, i.e., E(δ)>0. However, when σ² is large (but less than σ_(cut)²), a statistical test of whether E(δ) is greater than zero would allowa “margin of error” proportional to σ before such a declaration would beconsidered statistically significant. Therefore, in many standardstatistical inference applications, a resulting hypothesis test would bein the form of two cases. The first case is δ≦T, where it is concludedthat E(δ)=0 (or E(δ)≦0), and parity is achieved. The second case is δ>T,where it is concluded that E(δ)=Δ>0 (where Δ is a null or alternatehypothesis of the average difference, δ), and parity is not achieved.

T is a threshold that depends on σ and other test design parameters,including the “false alarm (FA)” limit, α, and the “miss (MS)” limit, β.By definition, the FA is the probability that the test declares a “fail”(i.e., δ>T), when in fact E(δ)=0 (or E(δ)≦0), and the MS is theprobability that the test declares a “pass” (i.e., δ≦T), when in factE(δ)=Δ>0. Mathematically, the FA and the MS may be written asFA=P[δ>T|E(δ)=0] and MS=P[δ≦0|E(δ)=Δ], respectively, with test designrestrictions that FA<Δ and MS<β. Therefore, the key parameters forchoosing the threshold, T, are α, β, Δ and σ. FIG. 6 shows a graphicalrepresentation of the relationships between these parameters, includingthe FA and the MS, with δ increasing along the x-axis in the directionof the arrow.

With respect to the relationships among the parameters α, β, Δ, σ andthe associated test threshold, T, for each set of parameters (α, β, Δ,σ), there may be many feasible thresholds (i.e., since constraints α andβ preferably provide bounds on FA and MS, not equalities) or no feasiblethresholds (i.e., the requirements are too strict). Keeping the FAlimit, α, small (i.e., making T larger) tends to raise the likelihood ofMS. Similarly, keeping the MS limit, β, small (i.e., making T smaller)tends to raise the likelihood of FA. Therefore, a reasonable objectiveis to preferably keep both α and β relatively small, i.e., less than orequal to 0.1. In addition, if the alternative hypothesis parameter, Δ,is very small (i.e., a few percent) it may be difficult to achievedesign objectives, since, for example, 97% and 99% average CRs may bedifficult to distinguish statistically. On the other hand, if Δ is verylarge (i.e., 5% or more), there is a risk of poor service since averageCRs of 94-96% might “pass” and achieve parity, when they should actuallyindicate service degradation and possibly trigger action by either theILEC and/or CLEC. As a result, Δ is preferably equal to 4, which is alsothe same alternative hypothesis and time-tested difference thatunderlies the action thresholds in the well-known equal-access arena(see, e.g., Committee T1 Report No. 11 (June 1991) and Bellcore SRSTS-000317 (September 1990)).

Feasible thresholds for a given set of parameters, α, β, Δ and σ, areall values of T that simultaneously meet the FA<β and MS<β constraints.This may involve either a numerical search, or, possibly, an analyticalsolution. Preferably, in the range of interest for α, β and Δ that waspreviously set forth above, T and σ_(cut) are computed with thefollowing equations:${T \approx {\frac{\Delta}{2} + {{s\left( {\alpha,\beta} \right)}*\sigma}}},{{{where}\quad{s\left( {\alpha,\beta} \right)}} = {0.232*{\ln\left( \frac{\beta}{\alpha} \right)}}}$σ_(cut) ≈ Δ * (0.184 + 0.923 * (α + β))It should be noted that a feasible value of T may only be calculated ifσ≦σ_(cut). Similarly, with these equations, when Δ is small, so isσ_(cut), which results in few, if any, feasible values of T. Inaddition, as Δ increases, Δ/2 becomes the dominant component of T, withan allowance of s(α,β)*σ for volatility in δ. The allowance ispreferably linear in σ, with a coefficient depending on 1n(β/α), and acutoff at σ_(cut). In fact, the allowance is positive, i.e., s(α,β)>0,only if β>α. Also, σ_(cut) is proportional to Δ, with a dominantcoefficient depending on α+β.

As stated above, α and β are preferably small, i.e., less than or equalto 0. 1, and Δ is preferably 4. The selection of preferred specificvalues for α and β will now be described Preferably, α≦β for tworeasons. First, while an ILEC may essentially choose to engineer in sucha way that β<<α or β≈0, that may not correspond to a realisticperformance assessment criteria, since it would report almost everyoccurrence of β>0 as a “fail.” Second, it may also have some othercounterintuitive implications. For instance, when β<α, the threshold, T,is a decreasing function of σ. This means that, in the presence of noise(i.e., variance, σ²), the parity test gets more conservative (i.e., asmaller T) in order to keep MS small. There should be some positive“allowance” for δ, however, because of inherent volatility around theaverage CRs when the day-to-day variation of δ (i.e., σ²) is relativelylarge. Thus, preferably β≧α, since T is then an increasing function ofσ.

Based on these two reasons, preferably 0.1≧β≧α. As previously set forthabove, the threshold, T, is most sensitive to the choice of Δ, andtherefore, there is much less sensitivity to α and β since, forreasonable parameter values, the “allowance” is only a small fraction ofT. For example, with a limit of β/α<5, the contribution of allowance isless than 20% of T, even for σ=σ_(cut). Consequently, for selectingspecific values, α and β are preferably based on well-accepted andwidely-used action thresholds known in the equal-access arena.Accordingly, by using the performance thresholds in the equal accessarena, the specific values for these parameters are preferably α=0.025and β=0.1, with β/α=4.

With α=0.025, β=0.1, and Δ=4, the above equations for computing thethreshold, T, and the SD cutoff, σ_(cut), may be rewritten as follows:T=2+0.322*σσ_(cut)=1.198FIG. 7 shows a graphical representation of these values and equations,and their use in Steps 206 and 208 of the parity metric of FIG. 5. Aswith Step 105 described above, the parity test “passes” in Step 208 whenthe average sample difference, δ, is less than or equal to thethreshold, T. The threshold, T, may be calculated by entering the SD, σ,into the equation for the threshold, T, set forth above. As shown inFIG. 7, the preferred threshold, T, ranges fairly linearly from 2.0,when the SD, σ, is zero, to 2.4, when the SD, σ, is equal to the SDcutoff, σ_(cut) (i.e., 1.198). If the parity test or metric “passes”(i.e., δ is below the line for T in FIG. 7), then a PASS 203 is declaredand the method 200 of the present invention is completed. In contrast,if the average sample difference, δ, is greater than the threshold, T(i.e., δ is above the line for T in FIG. 7), the parity test does notpass in Step 208, and a FAIL 209 is declared. Preferably, the FAIL 209is identical to the FAIL 107, with the same results.

It should be understood that the graphical representation of thesevalues and equations shown in FIG. 7 is not applicable to Steps 206 and208 of the method 200 of the present invention if the CR_(CLEC) is equalto or greater than 97%, since a PASS 203 would have already beendeclared in Step 202. Similarly, as shown in FIG. 7, for any δ less thanzero (i.e., negative) or equal to zero, regardless of σ, a PASS 203would have already been declared in Step 204. In addition, for ease ofillustration, the depicted values for δ and σ do not extend outside theranges (−1, 5) and (0, 2.5), respectively, in FIG. 7. It should beunderstood, however, that the actual values for δ and σ may extendbeyond these ranges.

FIGS. 8, 9A-9B, and 10 show several examples using the method 200 of thepresent invention, illustrating the possible outcomes of FAIL 209, PASS203, and INDETERMINATE 207 over a 20-day business month. In FIG. 8, twoexamples are shown to illustrate the FAIL 209 outcome for an ILEC andCLEC CNA. In example 1, the CR_(ILEC) is relatively high at about 99.4%,while the CR_(CLEC) is persistently and significantly lower than theCR_(ILEC) at about 73.2%. As a result, the average sample difference, δ,is 26.2, the SD, σ, is 0.286, and the threshold, T, is 2.092.Consequently, the parity test declares a FAIL 209, since the CR_(CLEC)is less than 97%, the sample difference, δ, is greater than thethreshold, T, and the SD, σ, is less than the SD cutoff, σ_(cut), of1.198.

In example 2 of FIG. 8, both the CR_(ILEC) and the CR_(CLEC) arerelatively high at about 97.8% and 95.2%, respectively, but theCR_(CLEC) is persistently and significantly lower than the CR_(ILEC). Asa result, the average sample difference, δ, is 2.6, the SD, σ, is 0.756,and the threshold, T, is 2.24. Consequently, the parity test declares aFAIL 209, since the CR_(CLEC) is less than 97%, the sample difference,δ, is greater than the threshold, T, and the SD, σ, is less than the SDcutoff, σ_(cut), of 1.198.

Examples 1-3of FIGS. 9A-9B illustrate the PASS 203 outcome for an ILECand CLEC CNA. In example 1 of FIG. 9A, the CR_(ILEC) is relatively highat about 99.1%, and the CR_(CLEC) is also relatively high at about98.3%. As a result, the average sample difference, δ, is 0.85, the SD,σ, is 0.314, and the threshold, T, is 2.1. Consequently, the parity testdeclares a PASS 203, since the CR_(CLEC) is greater than 97% (see Step202).

In example 2 of FIG. 9B, both the CR_(ILEC) and the CR_(CLEC) arerelatively low at about 74.5% and 73.2%, respectively, but the CR_(CLEC)is slightly lower than the CR_(ILEC) and somewhat more variable. As aresult, the average sample difference, δ, is 1.3, the SD, σ, is 0.259,and the threshold, T, is 2.083. Consequently, the parity test declares aPASS 203, since although the CR_(CLEC) is less than 97%, the sampledifference, δ, is less than the threshold, T, and the SD, σ, is lessthan the SD cutoff, σ_(cut), of 1.198.

In example 3 of FIG. 9B, both the CR_(ILEC) and the CR_(CLEC) are near100% at about 99.1% and 96.8%, respectively, but the CR_(CLEC) has twodays that are relatively lower (i.e., 85-90%) than the CR_(ILEC). As aresult, the average sample difference, δ, is 2.25, the SD, σ, is 0.86,and the threshold, T, is 2.3. Consequently, the parity test declares aPASS 203, since although the CR_(CLEC) is less than 97%, the sampledifference, δ, is less than the threshold, T, and the SD, σ, is lessthan the SD cutoff, σ_(cut), of 1.198.

In FIG. 10, two examples are shown to illustrate the INDETERMINATEoutcome 207 for an ILEC and CLEC CNA. In example 1, both the CR_(ILEC)and the CR_(CLEC) are relatively high (i.e., near 100%) at about 99.4%and 95.7%, respectively, but the CR_(CLEC) has one low day at about 60%.As a result, the average sample difference, δ, is 3.7, the SD, σ, is2.061, and the threshold, T, is 2.663. Consequently, the parity testdeclares an INDETERMINATE outcome 207, since the SD, σ, is greater thanthe SD cutoff, σ_(cut), of 1.198. Accordingly, the parity test is unableto determine whether performance parity is achieved for the ILEC andCLEC CNA.

In example 2 of FIG. 10, the CR_(ILEC) is relatively high at about99.4%, while the CR_(CLEC) is relatively low (i.e., 80%) for about halfthe 20-day business month period, with an overall average CR_(CLEC) ofabout 86.65%. As a result, the average sample difference, δ, is 12.75,the SD, σ, is 2.289, and the threshold, T, is 2.736. Consequently, theparity test declares an INDETERMINATE outcome 207, since the SD, σ, isgreater than the SD cutoff, σ_(cut), of 1.198. Accordingly, the paritytest is unable to determine whether performance parity is achieved forthe ILEC and CLEC CNA. It should also be noted that in example 2 of FIG.10, the cause of the disparity and lower CR_(CLEC) in the first half ofthe month appears to have been addressed and corrected, as evidenced bythe rise in the CR_(CLEC) to near 100% for the second half of the month.

The methodology of the present invention is particularly useful formeasuring performance parity among CNAs, since it may be summarized in afew lines of code, a table, or a simple “field of use” graph. It shouldalso be readily apparent from the foregoing description and accompanyingdrawings that the methodology of the present invention is an improvementover the prior art methodology of busy-hour trunk blocking for measuringnetwork performance parity. For instance, the method of the presentinvention is based on solid and classical statistical decision theorypractices, with assumptions that are consistent with widely-accepted andbroadly-implemented tariffs and performance measures in theinterconnection arena. In addition, the method of the present inventionutilizes innovative features that make it simple in structure, yetpowerful and flexible enough to discern among the three outcomes (i.e.,pass, fail and indeterminate). The method of the present invention isalso based on the call completion percentage or ratio measure, which isbetter correlated to realized quality of customer service than thetraditional trunk blocking of the prior art. Moreover, the method of thepresent invention will encourage good business behaviors among ILECs andCLECs because: (1) it “passes” only when both the ILEC and CLEC haveconsistently comparable CRs; (2) it “fails” when the CLEC's CR issignificantly and persistently lower than the ILEC's CR, therebypossibly leading to an investigation and to ILEC and/or CLEC networkaugments or upgrades; and (3) it has an “indeterminate” outcome thatpotentially results in a joint investigation and resolution of parityproblems in order to minimize unwarranted expenditures by either theILEC or CLEC.

Those skilled in the art to which the invention pertains may makemodifications in other embodiments employing the principles of thisinvention without departing from its spirit or essentialcharacteristics, particularly upon considering the foregoing teachings.Accordingly, the described embodiments are to be considered in allrespects only as illustrative, and not restrictive, and the scope of theinvention is, therefore, indicated by the appended claims rather than bythe foregoing description. Consequently, while the invention has beendescribed with reference to particular embodiments, modifications ofstructure, sequence, materials and the like would be apparent to thoseskilled in the art, yet still fall within the scope of the invention.

1. A method for measuring network performance parity, the methodcomprising: (a) determining a performance statistic for a networkprovider; (b) determining whether the performance statistic passes afirst test; (c) assessing whether a second test is determinative if theperformance statistic does not pass the first test; and (d) assessingwhether the performance statistic passes the second test if the secondtest is determinative.
 2. The method of claim 1, wherein the first testcomprises comparing the performance statistic to a calculated thresholdvalue.
 3. The method of claim 1, wherein the first test comprisescomparing the performance statistic to a predetermined threshold value.4. The method of claim 1, wherein the second test comprises comparingthe performance statistic to a calculated value.
 5. The method of claim1, wherein the second test comprises comparing the performance statisticto a predetermined value.
 6. The method of claim 1, further comprisingdeclaring a pass if the performance statistic passes one of the firstand the second test.
 7. The method of claim 1, further comprisingdeclaring an indeterminate outcome if the second test is notdeterminative.
 8. The method of claim 1, further comprising declaring afail if the performance statistic does not pass the second test.
 9. Themethod of claim 1, further comprising declaring a pass if theperformance statistic passes one of the first and the second test;declaring an indeterminate outcome if the second test is notdeterminate; and declaring a fail if the performance statistic does notpass the second test.
 10. The method of claim 1, wherein the performancestatistic comprises call completion ratio.
 11. The method of claim 1,wherein the performance statistic comprises a call completionpercentage.
 12. The method of claim 1, wherein the second test comprisesa parity test.